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ABSTRACT 

This paper describes the history, philosophy, 
cutcomes cf work at the Educational Technoloay Center at the 
Oniversitv of California, Irvine, with particular emphasis on the 
activities of the Physics Computer Development Project. Ten years 
evolution for the physics projects and its basis of grant support 
examined, and a series of principles is outlined as a guide for 
anyone working in the development of computer-aided learning 
materials. The outcomes of the project which are described include 
the preparation cf computer dialogs, the use of the computer for 
problem solving, the Irvine Center course management system, the 
design of full courses, and the Irvine authoring system. Background 
information on the evaluation procedure used for the Irvine project 
is also provided, as well as recommendations for the future of 
computer-based learning at the Irvine Center and in other educational 
settings. (MEB) 
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This paper describes the history, philosophy, outcomes and 
other aspects of the work centered in the Educational Technology 
Center at the University of California, Irvine. A major component 
of this activity has been performed by the Physics Computer 
Development Project. 
History 

It is difficult to know where to start describing our project. 
But I will take a personal point of view. My own interest in 
computers developed about 1959 vihen, as a faculty member at the 
University of Alaska, I attended IBM courses about a new IBM compute 
a 1620. This early, now seemingly primitive, machine was one of 
the first inexpensive machines to have a higher level language 
available, FORTRAN. It was not heavily used at the University of 
Alaska. I participated in the early use of it, working with a 
graduate student, Harold Leinbach, in developing a program which 
involves an electron coming through the atmosphere. A group of 
people stood around, and excitement was generated as to how far an 
electron would penetrate the atmosphere. 

Soon after that I began to teach a physics class in which I 
saw natural ways to use the computer and to make use of the 
excitement I had observed. This development continued for several 
years in Alaska until I left in 1962. 
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After Alaska I tried to determine to what extent such work was 
being done elsewhere. I finally received a letter from someone 
within IBM, later to become a friend, to the effect that they didn't 
know much about using computers in education but that someone in 
Alaska was doing it. At this point I decided that perhaps I was 
on to something new! 

After spending a yea.r as a National Science Foundation Faculty 
Fellow at Harvard, mostly working in the history of physics, I went 
to teach at Reed College for the next four years. At Reed College 
first a 1620 and then an 1130 allowed me to continue working with 
computers. I was fortunate to be teaching. a large natural science 
class for nonmajors, and the computer became a natural adjunct to 
that class. 

The Reed environment was limited to small computers. I made 
attempts to secure larger interactive systems in Portland, but that 
seemed to te in the future. Hence, when the opportunity was pre- 
sented to me (through Kenneth Ford, then chairman of the Physics 
Department) to join the faculty at the University of California, 
Irvine, I was happy to make the move. 

Irvine had committed itself already in its earliest stages to 
the importance of the computer as a learning device. It obtained, 
and still obtains, superior support for that purpose from the 
University of California systemwide. 

The Irvine situation was not entirely smooth. I arrived in the 
middle of a computer crisis. A large joint project between IBM 
and the University was developing a timesharing system for educa- 
tional purposes. The system was to be running when I first arrived 
on the campus, but early tests showed problems. This and various 
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other factors, including resistance from xt users, led to a 

hectic, first year. At the end of that year tl- ■ ill equipment left, 
and we had two new computers, a Sigma 7 and a PDP- 10. I played 
a considerable role in choosing the Sigma 7. 

About this time I received the first of a imber of grants 
from the National Science Foundation. Table 1 shows the grants 
received at Irvine. Existence of a good timesharing system, the 
Sigma 7, plus support from the National Science Foundation allowed 
me to do many of the kinds of things I had been planning. 



TABLE 1 

Grant Support from the National Science Foundation 



Dates 



Title 



Amount 



7-1-69 to 
12-31-71 



1-17-71 to 
9-30-72 



I- 1-72 to 
6-30-74 

6-1-74 to 

5- 31-75 

6- 1-75 to 

5- 31-76 

6- 1-76 to 

II- 30-77 

5-1-77 to 

4- 30-78 

5- 21-78 to 
11-31-81 



Development of Computer Based $350,426 

Materials for Teaching Undergraduate 

Physics 

College Science Improvement Program 56,200 

Cooperative Projects for Two-Year 

Colleges 

Computer Based Interactive Graphics 182,000 
in Teaching Undergraduate Physics 

Computer Graphics in Learning 213,100 



Computer Graphics in Learning 217,400 



Computer Graphics in Learning 230,433 



Conference on Educational Applications 17,550 
Intelligent Videodisc Systems 



A Testing and Tutoring Environment 
for Large Science Cotirse (CAUSE) 



249,200 



Total 



$1,516,309 
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As can be seen, our support from the National Science Foundation 
has been consistently good. The fact that we have been supported 
almost continually for a ten-year period has allowed us to make 
steady progress. Furthermore, it has led to increased use of com- 
puters within classes. This year, for example, we have beginning 
physics and beginning math classes (discussed further in a later 
section) in which all the tests are given on-line directly at the 
computer and with considerable additional material used in those 
classes too. These two classes do not cover the complete range of 
educational usage at Irvine; but much of what happens in other classes 
is similar to that to be found in other institutions. 

In addition to support from the National Science Foundation we 
have received several types of support from the University of California 
First, my department has been cooperative in allowing our work with 
computer-based learning materials to be my research activity at the 
University, This was agreed on before I reached the university, and 
I have no complaints with the way that agreement has been honored, 
even though the people who made the initial agreement have long 
since left. Second, through the Innovative Projects funds of the 
University, funds systemwide in the University of California that 
support curriculum development, we have had a series of grants. 
While these grants have been smaller than the National Science 
Foundation grants, they have been very helpful particularly in 
providing us capabilities for maintenance of programs. 
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TABLE 2 



Dates 


Title 


Amount 


1969/1970 


Development of Physics Room at 
at Irvine 


$ 12,000 


1972/1973 


Development of Motivational and 
Remedial Units for Use in Beginning 
Physics Class 


10,000 


1973/1974 


Development of Additional Materials 
for Physics 1 


3,450 


1974/1975 


Developing Computer Dialogs 


6,000 


1974/1975 


Uses of Irvine Physics Dialogs on 
Other UC Campuses 


30,250 


1974/1975 


Computer Programs for Instruction of 
of Medical Studies (with Hall and 
Ballard) 


9,400 


1975/1976 


Further Development of Instructional 
Material - Physics 3 


3,792 


1978/1979 


Computer-Based Learning for Beginning 
Physics 


10,448 




Total 


$ 85,340 



Another major contribution from the University of California 
has been computer time. The University receives funds directly 
from the Legislature for instructional uses of computing. Irvine 
has used each year some of these funds for developmental purposes 
as well as the more standard use of them for class work. Our 
project has benefitted from these funds. 

During the more than ten years that the project has been active 
at Irvine a number of people have played critical roles. Richard 
Ballard joined the project in 1970 as a codirector. Joseph Marasco 
joined the project in 1975 as another codirector. Both have now 
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left the project; Joseph Marasco still works with us in the design and 
development of materials even though he has a full-time job elsewhere. 
Our senior programmer for many years was Estelle Warner. More re- 
cently the people directly involved are Donald Darling, codirector 
of our CAUSE grant, from the Math Department at Irvine and Stephen 
Franklin, project director of the CAUSE grant , from the Computing 
Facility. 

Much of the work of the project has depended on a very devoted 
and hard working group of student programmers, almost entirely under- 
graduates, who have worked with the project for many years. Further- 
more, visitors, often for extended periods of time, have made major 
contributions in helping us write programs and in their overall 
design. 

Philosophy 

As with any long and extended project, one would expect some 
changes in the philosophical positions of the project. But I like to 
view these as evolutionary changes, consistent with our earliest 
philosophy but offering some refinements. 

The best source of information about the early philosophy came 
in speeches given by the project director, Alfred Bork, at the Second 
Annual Conference on Computers in the Undergraduate Curriculum at 
Dartmouth in 1971 and at a SIGUCC meeting two years later. The 
Dartmouth speech was one of the three keynote speeches at the meeting. 
I outlined a series of principles which I felt should guide anyone in 
the development of computer-aided learning materials in the form of 
six "theses." 

Thesis I : Different students learn in different ways. 

Most teachers would agree, but practically all courses seem to 
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ignore student differences. Most courses are rigidly structured, 
with only one path to success and only one set of learning materials. 
An environment truly responsive' to students must have a variety of 
materials and techniques for learning. 

Thesis_2I: We are only beginning the task of learning how to use 
the computer in education. 

I worry greatly about teachers who feel that they already know 
all the answers. We have a long way to go, and theoretical analysis 
will not tell us how to employ computers effectively. Hence we want 
to maintain flexibility, and we should be prepared for long years 
of trial and error while uoing computers in learning. 
Thesis III : Useful ways to involve computers in teaching may depend 
on subject matter involved. 

What is highly effective in physics may turn out to be useless 
for literature. This is obvious with computational uses, which are 
tailored for a specific need. It is perhaps less obvious with other 
types of usage. While we may find techniques which transcend subject 
matter boundaries, we will continue to need specific techniques for 
individual areas. 

Thesis IV: It is wise to retain all usage modes for computers in 
every learning situation. 
Computational, tutorial, simulational , managerial, and other 
modes as yet unnamed may all prove to be of great importance in 
in education. Further, in Mew of Thesis III, a mode worthless 
in one discipline may be valuable in another area. 

Thesis_V: We should continue to develop ways of learning independent 
of computers. 

Some areas and some students may profit from other techniques. 
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Many powerful learning tools exist. The film can be used inter- 
actively with computers as a very effective learning medium. Further- 
more, in some areas, nothing competes in teaching effectiveness with 
a student's experience in working problems. Thus, I am not in 
sympathy with the "whole course" approach to computers in education, 
where the computer becomes the sole medium. Today we don't know 
enough to use computers exclusively, and we may never want to. 
Thesis VI: The test of all learning is with students— does the 

material, computer or otherwise, lead to some type of 

learning for some students? 
All educational materials need to be widely used and continually 
modified on the basis of students' experiences. 

The subtitle of the speech also reflected the project philo- 
sophy. It was "Let a Thousand Flowers Bloom." The emphasis was that 
at that time we were very much in a learning mode, trying to under- 
stand how to use computers effectively in learning environments. 
Therefore the important thrust was to trj; many approaches, many dif- 
ferent modes of computer usage, rather than to focus early on a 
single approach. At the present time I believe we (meaning everyone) 
are still in this experimentation period, but approaching the end of 
that period. 

Another early "philosophical" paper from the project (first 
given at the SIGUCC meeting at the fourth CCUC) described "ten 
widely believed myths." The following is abstracted from that 
dociament . 
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Myth 1: You must choose between direct and adjunct use of the 
computer. 

Either students can do their own programming, using the computer 
as an intellectual tool— sometimes called the adjunct use— or students 
can interact with teaching programs prepared by others— the direct 
or mainline use. 

M2th_2.: You must have massive equipment to use the computer in' 
education. 

Some of the more interesting teaching applications have come 
from schools with minimal computer equipment. Small stand-alone 
minis certainly do rule out some of the kinds of things that can be 
done. Thus, dialogs are not possible on small minis; but many other 
types of usage are possible. 

Mzth_3: One language is much easier to learn than another. 

When students write programs for problems in physics or mathe- 
matics courses, the questions of which language to use and how 
students learn that language become important. Arguments are often 
based on supposed easy learning of one language or another; thus 
proponents of BASIC often claim that it is very easy to learn. My 
own experience indicates that the way the language is taught is much 
more important than the language itself in determining speed of 
initial learning. 

Myth 4: Computers will be widely used in education in present 
organizational structures or institutions. 
The ability to provide learning materials at any time and at 
any pace, and provide self-testing features, the ability to respond 
individually to students, to have access to large amounts of data, 
all imply that the way schools operate are likely to change drastic- 
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ally when computers are widely used in learning. One interesting 
view of how this revolution might happen is presented in Education 
and Ecstasy by George Leonard. 

Mzth_5: At this time your best buy for a terminal is the Model 33 
Teletype. 

I feel strongly that educational users should never buy Model 33 
Teletypes today! 

M2th_6: Computers are too expensive to use in teaching. 

Regardless of what one thinks about the costs today, the future 
situation is clear: Of all the costs involved in the educational 
process, computer costs are almost the only ones going down. Teachers, 
books, buildings, and films are going up in cost, while computer 
costs, because of a rising curve of technological development, are 
still diminishing dramatically; so the computer will become more 
and more competitive as a teaching device over the next few years. 
Mxth_7: If we acquire a CAI language, that solves our problems. 

The availability of a dialog language, no matter how good the 
language may be, is only a small part of the process of getting 
reliable and educationally useful teaching materials on the computer. 
The whole problem of an authoring system—the way one persuades 
teachers to write materials, the full facilities provided, the in- 
centives for doing this, the use of secretaries, program-iers , and 
other kinds of auxiliary people, the testing procedures, the gath- 
ering of feedback, and the preparation of suitable computer-related 
text material—is enormously more important than the question of the 
language itself. 

Myth 8 : PLATO and MITRE are solving all the problems. 

Both are very interesting projects. To regard them as exhaust- 
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ing all the possibilities, however, is quite wrong. Many interesting 
teaching materials that exist today could not be run on either of 
these systems. 

M2th_9; Valid educational material can be developed without involving 
experienced teachers in the area. 
Teaching is still teaching whether done by computer or by any 
other device. My experience shows that really effective educational 
materials are still coming, in spite of talk to the contrary, almost 
entirely from those who are very much involved in the teaching process. 
The intellectual structure of every discipline is different, and the 
tough question of fundamental goals cannot be resolved in any simple, 
quick way. While computer scientists and educational psychologists 
can help develop learning material, I do not believe they can do it 
alone. 

Mvth_J^: The computer used educationally uses only minor amounts of 
computer resources. 
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I can find little to quarrel with even today! Perhaps in 
Myth 2 I underestimated the future potential of small computers. 

The strong emphasis on graphics in the early days of the project 
has continued. Our earliest grant emphasized graphic capabilities. 
My feeling has always been that visual information plays an extremely 
important role in the learning process, and that every learning 
approach, whether involving computers or not, could normally expect 
to use graphical information. 

This belief in graphics has been reinforced and deepened by the 
experiences of the project. But graphics has r -ne to take a larger 
meaning than it had at first. Initially I had in mind only the 
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question of adding pictorial information. While this still plays a 
critical role, we now more and more consider text itself to be a 
graphic component. We consider the composition of each screen seen 
by students to play a major role in the effectiveness of the material. 
I will have more to say later about the implications for the importance 
of visual design in the developmental process. 

The fact that the project has never had any doctrinaire attitude 
toward the way the cjomputer could be used in education, but has 
always argued that many different ways might be usable, has had 
important ramifications. As we have moved along, we have often been 
guided by the necessities of classes. One of the dominant aspects of 
our work in later years, for example, has been the use of inter- 
active, on-line testing. These tests, to be described later, are 
combined learning and testing materials offering much help to students 
who are in trouble. But we came into this approach only gradually as 
we started to teach large classes with our computer-based material and 
encountered problems. The development of such approaches was com- 
pletely consistent with our previous philosophical position. 

Another shift during the course of the project in philosophical 
outlook was from developing individual pieces of material toward 
worrying more about how these materials fitted into a total cpurse. 
Initially it was the compelling examples that were of most interest 
to us. Often compelling examples turned out to mean compelling to 
us, to our evaluators, and to our many visitors rather than to the 
students. Some of the compelling examples were real bombs with 
students! This pushed us more into looking at the total class 
environment and to asking what was really needed for that. 



Another shift in philosophical position came about through 
external developments. When the project bejgan an interactive system 
almost certainly had to be a timesharing system.. So naturally, it was 
that type of hardware that was assumed. The rise, particularly in 
the last several years, of inexpensive, powerful stand-alone systems 
has completely altered the picture. Stand-alones offer, we argue, 
tremendous advantages in marketing and delivery of computer-based 
learning materials. Hence, our recent work is focusing in that 
direction. While this does not affect any of the underlying peda- 
gogical philosophy, it does very much affect our attitude toward 
transportability and related factors. 

One philosophical direction in which the project has differed 
from many other projects is that with regard to the use of special 
''CAI" languages. The decision was made in the ea? ly days of the 
project that as all the ways of using the computer educationally were 
not known we would not tie ourselves down to any CAI language. 
Rather, we would try to work within available general purpose lan- 
guages, exploiting the full capabilities of the computer. Our feeling 
was that as we gained more experience we would want to do things that 
we could not conceive of initially, so to begin by designing a 
language would be a mistake. We have found this to be a very effec- 
tive position and have maintained it. At the present time we are 
moving away from much of our present software into a new PASCAL-based 
system. The notion of working within very powerful, general purpose, 
higher level languages still seems to us to be the best way of pro- 
ceeding; we do not believe that any of the existing CAI languages 
will survive. 
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Outcomes of the Pro.ject 

I will discuss the outcomes of our work under five different 
headings — the preparation of computer dialogs, the use of the computer 
for problem solving, our course management system, the design of 
full courses, and the Irvine authoring system. 

1 . Student-Computer Dialogs 

The major continual activity of our project has been the prep- 
aration of ' interactive student-computer dialogs, learning conversa- 
tions betw^^n the people who prepare the material and the student. 
These programs typically take somewhere between one-half and one and 
one-half hours of student time for completion. They are often run 
in a number of sessions and can at the authors' discretioii allow 
restart at the point reached in the previous session. Some programs, 
particularly those which provide student facilities, are more open 
ended as to the amount of time that might be involved. Although most 
of our dialogs have been written by members of the project, visitors 
often participate in this process, so many others have names attached 
to them. 

For convenience I will discuss the dialogs in terms of a variety 
of categories. They cover a wide spectrum of pedagogical uses, not 
any single strategy. These categories, as with all human classi- 
fication systems, furnish no absolute distinctions and often categories 
are blurred. Further, several aspects often occur within a single 
program. 

An important early concentration was in interactive proof 
dialogs. The notion was to let the students develop important proofs 
in the course in as interactive a fashion as possible. This is to be 
contrasted with the student seeing the proof as done by the instructor 
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on the blackboard or as displayed in a textbook, a passive process. 
It was felt that students could better develop capabilities of doing 
such proofs themselves if they played an active role in the process. 
Many of our early dialogs were in this tradition, and some later 
examples can be found too. The earliest dialog, CONSERVE, written 
by Noah Sherman at the University of Michingan and the author, is an 
example. A more modern example is the MOMENTUM dialog, again a joint 
product of the author and Robert Eisberg of the University of 
California, Santa Barbara. 

A second class of dialogs is concerned with helping students in 
developing concepts . Again, the strategy is to avoid the author- 
itarian approach of many classes where concepts are introduced by 
fiat. We attempt to convince students that some reason or necessity 
exists for a particular concept before the name of that concept is 
introduced. Thus, in the dialog 3D (Arthur Luehrmann and Alfred Bork) 
the notions of divergence, curl and gradient are arrived at as gen- 
eralizations on the concept of derivative before the student has the 
names for these quantities. The FIELD program introduces slowly, with 
some use of simulation, the notion of electric field, again in an 
interactive fashion. 

A similar collection of dialogs is concerned with helping students 
with formal reasoning capabilities. It is- assumed that these students 
are not full formal reasoners yet, but are reasoning in what Piaget 
would refer to as concrete modes. This work has been particularly 
guided by our collaboration with Arnold Arons in the Physics Depart- 
ment at the University of Washington. Six programs have been written 
jointly with Arons and the author; Joseph Marasco was also involved in 
one of these programs, HEAT. Three of the programs concern magnetic 
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fields, and two are astronomical — the sky as seen from the earth and 
the phases of the moon. 

Laboratory simulations are also useful. We can provide an 
experimental situation for students similar to experiments that might 
be carried out in laboratories. But many more experiments can be 
carried out in a short period of time, and students can be immediately 
queried as to the meaning of the results. Perhaps the most extensive 
dialog of this kind is HOCKEY, again primarily due to Robert Eisberg. 
The experiments involve colliding pucks, first in one dimension and 
then in two dimensions with three types of collisions considered. The 
aim is to get students to arrive at the notions of mass, momentum, and 
conservation of momenttun. 

Controllable worlds are a type of simulation too but with empha- 
sis on a broader capability than those provided in the laboratory 
environment. The student is given the ability to manipulate the 
world, often in ways not usually available. The first controllable 
world, MOTION, provided a flexible f = ma world with the student 
picking force laws, initial conditions, equation constants, and what 
to plot. A second approach was in the FIELD dialog with the ability 
to plot electrostatic field lines and equipotential lines. 

But some of our attempts to produce extensions of this idea were 
not successful. We tried to produce a range of such simulations that 
provided much greater flexibility in interacting with students. But 
in spite of considerable project resources, these programs did not 
succeed. Only one reached running form, and this had a limited 
subject area. 

The possibilities with on-line testing were first explored by 
Stephen Franklin within the precalculus mathematics course. There it 
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was found that the tests gave a very natural way to conduct any 
personalized system of instruction course, relieving the instructors 
of the routine burdens of testing and record keeping but still providing 
quick feedback to students as to what problems were correct and which 
incorrect . 

After our initial experiences with tests we began to become more 
impressed with the possibilities, particularly in combining within 
tests some of the same types of learning capabilities that we had been 
providing in the other types of programs discussed. The intriguing 
possibility was to be able to offer direct and immediate assistance 
to students, finding out just what the student errors were and re- 
sponding immediately with pertinent help. The twenty-seven on-line 
mechanics tests were developed with this in mind. They cover the 
first quarter of a beginning science-engineering majors physics course 
(several different tracks, as discussed later) and represent the 
most extensive coherent development we have undertaken. 

These examples do not cover all the possibilities, but they give 
a good overall impression of the types of materials we have produced. 
A full list of dialogs is available from the project. 

2 . Student Problem Solving 

Our project has always felt that one of the most important 
aspects of the use of the computer within courses was in providing 
the ability for the students to use this powerful intellectual tool to 
solve their own problems. The computer thus ^'expands" the human 
intellect. This use of the computer often allows new approaches to 
courses, approaches impossible before the computer existed. 

One major question is what computer languages students should 
use for this problem solving activity. The project has developed a 



continually revised position paper with regard to this issue. It was 
felt from the beginning that the issue of languages should be ration- 
ally decided! That is, we should be able to set criteria as to what 
was desirable, and then to compare existing languages against these 
criteria. The result was that the two choices that are ''besf for 
contemporary science-engineering students are APL and PASCAL. The 
arguments for each are very different. Our paper argues that the 
least desirable language for this purpose is BASIC. In most cases 
this issue is decided either simply on what is available or on grounds 
of sheer emotionalism. 

In addition to picking the right language another important 
overall issue is that of efficient methods of learning the language. 
If a computer language is to be learned as a part of another course, 
such as a physics course, it must be very efficient in terms of student 
time. Traditionally university courses have not been that coHvOerned 
with efficient use of student time. Often it is the professor's time 
that is being economized! But student time is an important issue. 

We have found two generally effective ways of teaching languages; 
both can be used with some languages but not others. The first 
approach is to start with whole programs, to explain these programs, 
and then to have the students make modifications of them. The second 
approach, which we have used extensively in APL, is the ten finger 
approach. Here the students sits down, gets into APL, and is given 
(in separate printed material) a series of lines to type in. By 
observing the results of these lines a controlled discovery process is 
generated. 
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3 . Course Management 

It is universally recognized that a major problem in any large 
course is the bookkeeping problem, the recording of grades, the 
compilation of final scores, and related activity. Many partial 
computer systems have been designed to help with pieces or all of this 
problem. Hence, as we moved more toward large courses as will be dis- 
cussed in the next section, we became more concerned with course 
management . 

We decided to design a general course management system with 
little restriction on the type of course. Thus, we wanted the same 
facility to work for both a traditional course and for a course 
taught through the personalized system of instruction or Keller plan. 
The course structure, including all the information about what things 
were within the course, were to be part of the database. 

Naturally an important component of any course management system 
is the recording of information. This can come directly from the 
on-line quizzes or through secretaries or instructors on off-line 
activities related to the course. 

Another important activity of a course management system, par- 
ticularly in a personalized system of instruction environment, is the 
control of the quizzes — the question of who can take what quizzes 
and. when. This will again be a function of course structure and 
can be determined by the author for any particular course. Some 
modules may depend on other modules and some may not . Also the 
question of what is to happen after a quiz is not passed is important. 
It may be desirable, for example, to impose a time delay after a 
student has not passed a quiz once before allowing the student to 
take the quiz over again. It may also bp desirable to impose a limit 



on the number of tries a quiz can be taken before the student must 
see the instructor. 

A final 'aspect of course management is the feedback aspect, 
providing various types of information to the instructor and to the 
student. Of particular importance we feel is providing flexible 
feedback to the instructor, so the instructor can spot problems that 
are developing in the class and can contact individual students based 
on this information. 

Our management system at Irvine allows all these variants and 
also provides feedback to aid the instructor in spotting potential 
course or student problems. 

4 . Full Courses 

Development of complete courses begins to put together some of 
the components already discussed, since many of them play a role. 
This development also brings in many factors, however, that have 
not been discussed. We have tried to use the computer to produce 
special types of courses difficult to obtain in other ways. 

The major emphasis is on providing greater choice for students. 
The notion was that students should have a variety of choices which 
were offered in few traditional courses. Tor example, the use of the 
PSI environment, although it is not unique to us, allowed students 
choice in pacing through the material. A choice in learning materials 
is also desirable. ' 

Another choice, much rarer, which we provide in our newly struc- 
tured course is that of providing different content within the same 
course. The on-line quizzes covered several different contents; no 
one student took all the 27 quizzes that were developed. The strategy 
could be extended to give several courses within the same administra- 
tive format and with the same staff. 
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In the development of a course much material is needed besides 
the dialogs themselves. Module descriptions were developed for 
each of the student tracks. For one of the alternate approaches a 
full set of notes, begun at Reed College, was further developed while 
at Irvine. A set of exercises for increasing intuition, used with 
the controllable world dialogs, was developed also. In general our 
philosophy to learning material is to provide a great variety of 
learning materials associated with each of the units. 

The two principal courses developed were the precalculus mathe- 
matics course, developed primarily outside the project but using the 
extensive software developed in the project, and the first quarter of 
the introductory physics course. 
Methods of Implementation 

One of the major interests of the project has been the develop- 
ment of an effective authoring system. By an authoring system I 
refer to the entire process of producing computer-related teaching 
materials. An authoring system should be friendly to the various 
people who work with it, particularly the instructors involved in 
the project. It should reduce costs as far as possible. It should 
lead to highly effective material. 

Older authoring systems developed almost without thought. They 
could be fairly described as cottage industry systems, where the 
approach was for one person to do all the work. The cottage industry- 
approach, while it may work for early experimental stages, is inap- 
propriate for any large-scale production activities. Furthermore, the 
whole "lore" that grew up with it, including a collection of special 
languages, will, we believe, have to be abandoned eventually. 
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The problem suggested a systems approach. An analysis of the 
process involved and a structuring of a system to meet the needs is 
the rational approach. 

One aspect of such an investigation was that we could conven- 
iently employ other kinds of models. The most suitable models seemed 
to be those already used for large-scale production of educational 
materials. Within academic situations the work at The Open University 
was particularly interesting, even though it did not refer directly 
to computers. Furthermore, the structure of the textbook industry 
also gives very useful clues about how to proceed. 

One important criterion of the authoring system from the earli- 
est stages was that it should not restrict in any way the types of 
materials produced. We felt unhappy with systems, such as ones that 
were proposed elsewhere, where the authors were put in procrustean 
beds to make some of the later stages of the process easier. We felt 
that the focus should always be on the pedagogical design, and the 
greatest freedom should be allowed at that stage. 

We gradually evolved toward seeing the authoring system as in- 
volving seven steps. in each of these seven processes different 
types of people will play the dominant role. Some steps seemed to us 
to be well understood in much of our current activities. Others we 
believe we understand, but are typically not in a position to carry 
out fully. In one case the process is one where much is to be learned. 

The first and focal part of the activity is instructional design 
for the development of the pedagogical specification of the materials 
that are to run on the computer. Since faculty members are presum- 
ably the experts in how to promote learning in their own subject 
areas, they must play a major role in this process. Educational 
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psychologists and other instructional design individuals may also be 
involved. We have found this activity to work best in groups of two 
or three, working intensely. Pleasant environments away from inter- 
ruptions in the usual university environments and away from telephones 
are a necessity. The work is not easy, because unlike ordinary 
teaching activities the teachers must design in effect a whole range 
of conversations at a single time and must look carefully at what 
kinds of student problems should be considered. The instructors need 
detailed acquaintance with students to be perceptive in the handling 
of these problems. They work by creating informal flowcharts, 
specifying all the decisions to be made, the text, and the drawings. 
I emphasize that this is the most important stage of the authoring 
process. 

The second stage in authoring, the editorial process, is the 
least understood part of our structure. Many book publishers are 
accustomed to a greater and greater editorial role. They may feel 
uncertain, about the present situation because they do not see a clear 
editorial role. In our recent activities we have found that the 
people who are familiar with the instructional design of materials 
can take other people's material and offer editorial comments. With 
the vocabulary the process is like any other editorial comment for 
book-like environments. But when it comes to the question of what 
decisions are to be made based on student input the issues become 
fuzzier. More experience is needed. 

The growing importance of visual design in our project has 
already been discussed when the project philosophy was considered. 
Early computer material almost entirely ignored this, letting things 
appear on the screen or page simply where the computer language chose 
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to put them. We have moved consistently toward a conscious control 
of the design process. 

The ideal person to be involved in screen design is a competent 
graphic designer, the type of person who might do layout for a 
magazine ad. For this designer to function effectively, it is 
necessary to develop specialized kinds of software. Our screen 
design software allows such an individual to sit at a computer screen, 
to compose a variety of elements, to move them around the screen, to 
change their size and shape in various ways, and then to tell the 
program to write the necessary code. .Both graphic and alphanumeric 
aspects are covered; a variety of textual treatments are allowed. 
Newer developments working with stand-alone systems with selective 
erase capabilities are bringing in such new possibilities as the 
ability to control scrolling and formatting within individual areas 
of the screen while not affecting the rest of the screen. 

The next component in the authoring syscem is that of logic 
design. This is the part that is most closely related to traditional 
computer programming activities. It is clear, therefore, that this 
work will be done by programmers, either professional programmers or 
those particularly trained for this purpose. We now work with under- 
graduate students. 

The question of the language environment for logic design must 
be considered. In the cottage industry stage very specialized 
languages were designed with the pious (but unsatisfied) hope that 
they would be easy to use by ordinary teachers. Such languages as 
COURSEWRITER, PILOT, PLANET, and TUTOR fit into this category. We 
felt, however, that any such language would limit the type of material 
that was produced, and we preferred to stay within powerful, general 
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purpose, computer languages, developing whatever special capabilities 
that^were needed within that environment. Thus, if we needed new 
resources, they could be constructed because we were working within a 
general purpose language. Our older work used a variety of such 
languages, a mixture of assembly language macros for the interactive 
parts, FORTRAN for calculat ional parts, APL for the interactive 
design materials just described, and COBOL for database management. 
Our newer work is proceeding within PASCAL as a single language in 
which to develop all aspects. 

After the program is initially prepared the next stage is for 
the individuals involved to sit down together and run the program a 
few times, criticizing it and suggesting changes. Things look very 
different when one sees them on the screen than they did when they 
were on paper, and many new elements come in through a series of 
reworkings of the program at this stage. 

Next, class use is essential. No matter how good the designers 
were the question of how the material works with a-ctual students is 
critical. More will be said about this in the evaluation section. 
An important point is that the computer can save, information and this 
can be a powerful aid in improving the dialog. 

The next obvious stage is modification of the program based on 
the information obtained. Again, the strategies in the above stages 
must be flexible enough to allow easy modification, often by pro- 
grammers and graphic designers other than those initially involved 
with the material. These last several phases may go through a number 
of cycles with modifications leading to new class use leading to new 
modifications. 
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Evaluation of the Irvine Project 

Two types of evaluation will be considered. First there are 
internal evaluation activities conducted by the project itself. 
Second are external evaluations conducted either with project funds 
or through independent agencies or individuals. 
Internal Evaluation 

Perhaps the major test of the Irvine materials has been their 
repeated use in large classes. Unlike some projects which develop 
materials that get used only in trial classes, we have aimed from the 
beginning at dialogs which can be used in large class environments in 
regular courses. We have always believed that this is the "true 
test" of the material. Use with a few students where project members 
can play a major role in working with the students (either intention- 
ally or unintentionally) does not show whether the materials can be 
used in general environments outside the 'developing institution. 
Our mechanics material has now been used for four separate quarters 
in two different types of beginning physics courses. About 800 
students have been involved. 

One of our major interests in working with large classes is to 
see whether the course restructuring ideas, the possibility of allow- 
ing students to choose content, is possible in typical large courses. 
In each of the classes we have offered a variety of choices with the 
choices differing somewhat from time to time. We have demonstrated 
that the instructor can decide for a particular course what the 
options should be and that students can be suf f icieiitly informed to 
make choices. 

We haye also shown that we can be responsive to students in a 
number of ways within the course and that ^ the overall system provides, 
therefore, a reasonable learning environment. 
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External Evaluation 

Several types of external studies of the project can be men- 
tioned. Perhaps the most elaborate of these was conducted by Michael 
Scriven with funds from our grants from the National Science Foundation. 
Two major aspects of this study were examination of the course itself 
while running and an examination of some of the controllable world 
dialogs. Participating in the study were Jill Larkin and Mark St. 
John from the Sesame program at Berkeley. The formative study 
tended to agree with our internal evaluations of where the courses 
were strong and weak. Perhaps the material which received the 
highest rating was the help sequences in the on-line quizzes. Students 
in fact felt deprived in quizzes that did not have help sequences. 
On the other hand, the material that still presents the greatest need 
for improvement is the material designed to increase physical intui- 
tion. Time has not permitted us to rewrite this material as yet; we 
received useful and specific advice from the evaluators in this 
regard. 

Another type of external evaluation could be described as peer 
review. A number of different aspects are interesting. We often 
"exhibit" at national meetings with gratifying results. The project 
has had a steady group of visitors from all over the world, and these 
visitors have often written glowing descriptions of the project in 
various publications. 

A somewhat more formal activity of this type occurred in con- 
nection with Change magazine's project to identify innovative teaching 
in the United States. This project was not particularly concerned 
with computers but rather with excellent courses. It was a 
discipline-oriented project. In each case Change went to the discpline 
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leaders of education and asked for a selection of projects. Change 
then made a preliminary investigation of these projects and selected 
typically four projects for major articles in each discipline. Our 
project was one- of the four chosen in physics (Change , January 1978). 

It is interesting to look at the list as a whole incidentally, 
because a fair number of computer-related developments occurred in 
various areas considered by Change . 
The Future of Computer-Based Learning 

It is important in a growing and dynamic area as computer-based 
learning to give some thought to the future. We should be able to 
suggest where developments should take place next to assure that this 
learning tool will develop as rapidly and as powerfully as possible. 
This is particularly critical in an area where large companies will 
soon be involved in marketing activities. 

The following topics indicate areas in which I believe research 
is important at the present time or areas of future development. 
We cannot do all of these within our own project. However, many of 
them do indicate directions that we would like to pursue. 

1 . Research in Learning 

In spite of all the work that has been done we still have much 
to learn about the learning process and about how the computer can 
be most effectively -used in that process. Hence, continued research 
in computer-aided learning is essential. We have only scratched the 
surface of the question of how the computer is most effective 
in learning. Even the issue of more effective production of material 
needs additional work. 

I do not believe, however, the assertion of some that all 
developmental efforts should stop until this research takes place. 
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That would only lead to poorer materials, because others, not always 
with views we might like, will continue to work. 
2. Stand-alone Systems 

I have argued in several articles that the future of computer- 
based learning will be very much dominated by intelligent stand-alone 
systems, systems which will only on rare occasions converse with 
another computer elsewhere. These systems will continue to decrease 
in cost and increase in sophistication for a long period. 

Of particular importance will be the increasing number of home 
computers. I have seen estimates that 250,000 personal computers 
have already been sold, and projections of a saturation market of 
60,000,000. Whether this is accurate or not, it is clear that this 
market will continue to grow as more and more large vendors enter it. 
Several important announcements in this direction are forthcoming. 

The personal computers of today are not entirely suitable for 
educational purposes. But they are improving rapidly. Most home 
systems employ graphics. It is particularly interesting to note 
that it is now the case that one can buy complete graphic system^ 
(stand-alone computers) for less than one can buy graphic terminals! 
The reasons for this are not entirely clear, but I believe they are 
at least partially due to the fact that the complete graphic systems 
are later designs. Older graphic terminals tended to be ordinary 
CRT terminals with a separate new component built into them to 
handle t^^ graphics. More recent stand-alone devices are integrated 
designs, so these new systems can realize cost savings because of 
this approach. 
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3. Intelligent Videodisc 

Another important factor with regard to hardware in the immed- 
iate future will be the development and use of what I have termed 
elsewhere the "intelligent videodisc" system. The idea is to combine 
in an intimate hybrid blend the capabilities of the stand-alone 
computer just mentioned and the capabilities of the new optical 
videodisc systems. The videodisc systems are not only now available 
for the industrial-educational market, but they are also available 
for the home market. Home systems are currently selling for under 
$700. 

The importance of systems for education which combine the com- 
puter and the videodisc is that they allow us to bring into the 
computer-based learning environment a full multimedia capability. 
Thus, at any point within a learning sequence a slide, a video 
sequence, or audio messages can be presented. All of these media 
and the computer code can be stored on the same inexpensive, long 
lasting disc. So the intelligent videodisc can extend our present 
computer-aided learning resources to these new dimensions. 

The market is essentially a mass market. The cost of making 
discs is expensive in small quantities but becomes cheap when the 
quantities are large. The discs are easy to handle and so form a 
marketable product. Several developmental projects in this direction 
are already proceeding, but much additional work is needed. The 
economic issue of computer alone versus computer plus videodisc must 
also be carefully investigated. 

4. Continual Improvement in Quality 

Much of the period of the last twenty years has gone into 
understanding how to use the computer as a learning device. In 
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addition to the research already mentioned part of this is simply 

the gaining of experience, the trying of many different possibilities. 

I believe that we still have a distance to go in this experience 
gathering. Although the materials now available are much better and 
much more effective than they were a few years back, they still 
represent only a glimmering of the future potential. Almost all of 
the current modules will look relatively primitive only a few years 
further along. I am not simply referring to any current materials 
but even to our best efforts to this time. Very little that is 
currently available will find its way into commercial release. 

5. Increased Use 

The amount of use of computer-based learning material in edu- 
cation at any level in the world is almost trivial at the present 
time. Even in universities that claim to be doing much of it, only a 
few courses are typically involved. The major delivery system in 
education worldwide is still the combination of lecture-textbook, 
starting at an early age and continuing through graduate school. 

In the next few years we will see a sharp rise of computer-based 
learning activity, as more effective materials and less expensive 
machines are available to the user. Increased use will be seen at 
all levels of education, and it will also be seen in traditional and 
newly developed nontradit ional educational modes. 

6. Further Developments in Authoring Systems 

The process of development of computer-based learning materials 
as we carry it out at Irvine has been described. Further studies of 
this process and further attempts at variants are essential for 
further progress. We need to refine our techniques so- that effective 
material can be produced at reasonable cost. Specific problems exist 
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as already pointed out. For example, the question of how to use 
editorial services within the process needs further experience. 

7 . Large-scale Production and Distribution 

Currently our production and distribution modes for computer- 
based learning material are almost nonexistent. A few organizations 
such as CONDUIT are distributing a very limited range of materials. 
All the marketing mechnisms and production and distribution mech- 
anisms that exist for established educational media such as the 
textbook do not exist for computer material. Thus if a mathematics 
professor is looking for computer-based learning material, either 
part of a course or an entire course, there is simply no way to turn 
at the present except in very limited areas. 

As usage increases and as the commercial vendors see a larger 
and larger market in the home and in schools, we can confidently 
expect that more material will be available, and that national market 
ing activities will begin. Interesting questions arise as to which 
kinds of companies will be involved in this process. Clearly a 
number of existing companies have a considerable stake in the market. 
One is the textbook publisher. Another is the computer hardware 
manufacturer. 

But new companies may be involved. Several such companies have 
already come into existence, and more may appear as the older com- 
panies find that, as is typical, it is hard to adapt to new condition 

8. Changes in Traditional Institutions ^ 

As some traditional institutions begin to use the computer more, 
questions will arise as to how these institutions will change, and 
questions will also arise as to what will happen as other institu- 
tions, because of their conservative policies, resist change. 



The computer allows new educational possibilities different from 
those in current use. It allows self -pacing not only within in- 
dividual courses but within the entire curriculum. There is no 
reason why courses should start on quarter, semester or term boun- 
daries or why students should spend a fixed amount of time in either 
individual courses or in the entire educational program of the insti- 
tution. With new and powerful learning resources available around 
the clock and with these resources being individualized so each 
student can move at his or her own pace, we can expect changes. 

Along with these scheduling changes will probably also come 
changes in the grading system. Many studies have indicated that 
current normative grading has almost no correlation with any aspect 
of the student's life after school. While the possibility of grading 
by competencies completed is new (and so little evidence is available 
concerning its connection with the real world), at least it provides 
new opportunities to make success in school relevant to success in 
society. 

We can expect that some institutions will resist changes and 
will regard the computer as the work of the devil. The interesting 
question will be whether these organizations will continue to remain 
competitive, as students become more and more aware of the effec- 
tiveness of computer-based learning and as society becomes more and 
more concerned with the costs of traditional education. 

9- Changes in the Role of Teachers 

Teachers are very much concerned with the question of how the 
computer will affect their status. One obvious answer is that the 
large-scale production of computer-based learning material, already 
mentioned, will require very large numbers of teachers. Hence, many 
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teachers may shift from the deliverer of courses to the designer and 
developer of materials. 

A second change in the role of the teacher will be to handle the 
learning problems which the computer and other learning media avail- 
able cannot handle. There will always be need for this, for the 
teacher using expertise to determine why the student is not learning 
in spite of a rich variety of available modes and approaches. I 
already begin to see such a change developing in my own experiences 
with classes that depend heavily on the computer. This will mean 
that the teacher will spend less time in large group environments and 
more time with individual students. 

10. New Institutions 

Perhaps the most exciting development in the period ahead will 
be the rise of new types of educational institutions, some resembling 
such new existing institutions as the Open University in UK. Many 
of these new institutions will be able to capitalize greatly on the 
computer as a learning device, particularly as the computer becomes 
more and more available in public libraries, other public places such 
as science museums, shopping centers, airline terminals, and homes. 
Others may come out of such areas as extension services of existing 
institutions, while other institutions may be entirely new. 

11 . New Educational Systems 

The rise of new types of institutions, the demise of some ofy the 
more traditional institutions, plus the changes in other formal 
institutions imply massive changes in the educational system, a new 
system. This is perhaps the most difficult of the changes to foresee, 
since it depends on so many other factors already introduced. This 
development will be the capstone of the entire movement. 
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